{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jermwatt/machine_learning_refined/blob/main/notes/9_Feature_engineer_select/9_1_Intro.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chapter 9: Principles of Feature Engineering and Selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook contains interactive content from an early draft of the university textbook <a href=\"https://github.com/neonwatty/machine-learning-refined/tree/main\">\n",
    "Machine Learning Refined (2nd edition) </a>.\n",
    "\n",
    "The final draft significantly expands on this content and is available for <a href=\"https://github.com/neonwatty/machine-learning-refined/tree/main/chapter_pdfs\"> download as a PDF here</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9.1  Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this Chapter we discuss a number of principle facets of both *feature engineering* and *feature selection*. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9.1.2  Feature engineering "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature engineering methods consist of an array of technicques that are applied to data *before* they are used by either supervsised or unsupervised models.  Some of these tools - like the *feature scaling* techniques described in Sections 9.1 through 9.3 - properly *normalize* and cleans input data.  This provides a consistent preprocessing pipeline for learning, and drastically improves the efficacy of zero / first order local optimization methods in tuning machine learning models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another branch of feature engineering focuses the development of data transformations that extract the *maximal* amount of useful information from input data.  For example in the case of  a two-class classification, these tools aim to extract critical elements of a dataset that ensure instances within a single class are seen as ”similar” while those from different classes are ”dissimilar”.  Designing such tools can require significant domain knowledge - or in other words, a rich set of experiences dealing witth particular kinds of data.  However one simple concept - the *histogram* feature transformation - is a widely used tool for employed for this task across a variety fields for machine learning problems involving an array of data types - including categorical, text, image, and audio data.  We give a high level overview of this popular approach to feature engineering in Section 9.4. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9.1.2  Feature selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Human beings can be an integral component of the machine learning paradigm, and in practice it can be crucial that individuals be able to interpret and / or derive insights from a fully machine learning model.  The *performance* of a model is a common and relatively easy metric for humans to interpret - i.e., does the model provide a good predictive result (where 'good' can mean it e.g., the learner achieves an agreed on benchmark value).  It can also be very useful to understand which *input features* were the most pertinent to achieving solid performance (remember each dimension of input is called a *feature* or *input feature* in the parlance of machine learning).  Doing this helps us refine our understanding of the nature of the problem at hand and - depending on the application - can help us make *informed decisions* based on the results of *supervised learning*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Chapters 5 - 7 we saw how the fully tuned linear model for *supervised learning* takes the form \n",
    "\n",
    "\\begin{equation}\n",
    "\\text{model}\\left(\\mathbf{x},\\mathbf{w}^{\\star}\\right) =  \\mathring{\\mathbf{x}}_{\\,}^T\\mathbf{w}^{\\star}   = w_0^{\\star} + x_{1}^{\\,}w_1^{\\star} + x_{2}^{\\,}w_2^{\\star} + \\cdots + x_{N}^{\\,}w_N^{\\star}.\n",
    "\\end{equation}\n",
    "\n",
    "where the weights $w_0^{\\star},\\,w_1^{\\star}\\,...,w_N^{\\star}$ are been tuned by minimizing an appropriate cost function like e.g., the Least Squares when solving a regression problem of the Softmax cost when dealing with classification.  Understanding the intricate connections input features correspond with output naturally boils down to human analysis of the model's $N+1$ tuned weights.  However human beings often struggle to derive meaning from $N+1$ numbers (the value of all these weights) taken in all at once, and the idea of human interpretability quickly becomes untenable as $N$ grows."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To aid in this human investigation we often perform what is called *feature selection* which helps highlight the weights corresponding to the most pertinent features of a, i.e., those that lead to strong performance.  In this Chapter we discuss two popular ways for performing feature selection: boosting and regularization."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "navigate_num": "#000000",
    "navigate_text": "#333333",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700",
    "sidebar_border": "#EEEEEE",
    "wrapper_background": "#FFFFFF"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "86px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
